Calculating and Using Confidence Intervals for Model Validation

A confidence interval is an interval (i.e., a range of values) estimate of a parameter of a population (e.g., a mean) calculated from a sample drawn from the population. A confidence interval has an associated confidence level, which is frequency with which a calculated confidence interval is expected to contain the population parameter. Confidence intervals are of interest in modeling and simulation because they are often used in model validation. Typically, a set of executions of the model to be validated, which is a sample from the population of all possible executions of the model, are run and from their results a confidence interval is calculated as an estimate of the population parameter (e.g., mean model output value) that would result if all possible model executions had been run. Then, if the corresponding known or observed value for the simuland is within the confidence interval calculated from the model executions, or within some acceptable tolerance of the confidence interval’s endpoints, the model is considered to be valid for the parameter in question. This paper is an introductory tutorial and survey on confidence intervals in model validation. Confidence intervals are introduced in a statistical context, their interpretation and use in model validation is explained, and examples of the application of confidence intervals in validation are presented.